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re*a l-time or continuous data strea m s, such as real-t ime 
multimedia streamsTe.g. f live audio and video clips. Mul- 
timedia servers are grouped into two or more sets, for 
example wherein a first set includes one or more primary 
servers using odd-numbered ports and a second set includes 
one or more secondary servers using even-numbered ports. 
The client requests a multimedia stream through a control 
server or gateway which routes requests to the multimedia 
servers; and the client receives the stream directly from a 
selected (primary) server. T he client automatically detect s 
lo ad imbalances and/or failures (com ply partjal) and 
dynamically switches to a secondary server in order to 
continue receiving the real-time multimedia stream with 
minimal disruption and while maintaining a balanced load 
across multiple servers in a distributed network environ- 
ment. The determination can be made based on: the received 
bit or frame rate (for video); a bit rate or sample rate (for 
audio); monitoring a delivery rate or for packets arriving out 
of order: for example using packet numbering mechanisms 
available in TCP; sequence numbering or time stamp capa- 
bilities of RTP (in combination with the User Datagram 
Protocol (UDP)). In any case, the determination could be 
based on the rate measurement or monitoring mechanism 
falling below (or exceeding) some threshol d. Alternately , the 
primary server or t he control server could *send an exp licit 
distress or ftwiicn si£nal to the client. A n explicit signal can 
be used for example to switch clients in phases with minimal 
disruption. 

46 Claims, 9 Drawing Sheets 



1.8 



Client 
Agent 



,17 



1.1 - 



Server Architecture 



Control Server 




06/15/2002, EAST Version: 1.03,0002 



US 6,195,680 Bl 

Page 2 



OTHER PUBLICATIONS 

"desktop conferencing & collaboration", http://www.wpi- 
ne.com/, May 26, 1998 White Pine Software, Inc., 17 pages. 
"CU-SeeMe Reflector", http://ww.e92plus.co.uk/reflec- 
tor.htm, Enhanced CU-SeeMe, Feb, 3, 1998, 2 pages. 
Michael Sattler, "Internet TV With Cu-Seeme", Published 
by Sams Sep. 1995, Abstract from: http://www.amazon- 
xom/exec/obidos/ISBN~1575210061/ 
6156-9453266-197205, Feb. 3, 1998, 2 pages. 
Vivo Software, Inc-Home Page, http://www.vivo.com/, 
1997 Vivo Software, Inc, May 27, 1998, 4 pages. 
Download RealPlayer Plus, http://www.real.com/products/ 
playerplus/index.html?src=download, Copyright (c) Real- 
Networksjnc. and/or its licensors, 1995-1998, 4 pages. 
InterVu:Home, http://www.intervu.coin/, May 26,1998, 
InterVu, Inc, 3 pages. 

Video and Audio: Organization and Retrieval in the WWW, 
http://www.vosaic.com/corp/papers/www5.html, Z. Chen et 
al., 17 pages, Jul. 12, 1997. 

VOSAIC Corp.Home Page, http://www.vosaic.com/, Copy- 
right (c) 1997-98 Vosaic LLC, 11 pages. 
StorNet Texas Home Page, http://storl.stornet.com/Wel- 
come.html, Mar. 24, 1998, 5 pages. 
Marathon Products, http://www.marathontechnologies.com/ 
products.htm, "Continuous Uptime for Windows NT Appli- 
cations", Mar. 24, 1998, 2 pages. 

Software Fault Tolerance, http://www.marathontechnologi- 
es.com/software.htm, Marathon's Endurance™ Technology 
& Software Fault Tolerance, Mar. 24, 1998, 5 pages. 



Stardust IP Multicast Initiative — Release, http://www.ip- 
multicast.com/press/pr980407.hmtl, Television and Internet 
Multicasting Differences Clarified, Apr. 7, 1998, 3 pages. 
MBone(or IP Multicast Information Web, http://www.m- 
bone.com/®1994, 1995, 1996, 1997 Vinay Kumar, 26 pages, 
D, M. Dias et al., "A Scalable and Highly Available Web 
Server", COMPCON '96, 8 pages. 

T Brisco, "DNS Support for Load Balancing", Rutgers 
University Apr. 1995, Network Working Group Request for 
Comments: 1794, Category: Informational, 6 pages. 
P. Mockapetris, Domain Names — Implementation and 
Specification, ISI, Nov. 1987, Network Working Group 
Request for Comments: 1035, 52 pages. 
C R. Attanasio et al, "A Virtual Multiprocessor Imple- 
mented by an Encapsulated Cluster of Loosly Coupled 
Computers", IBM Research Report, Oct. 26, 1992, 2 cover 
pages and pp. 1-13. 

VDOStore-VD Olive Player.Free video streaming client, 
http://www.clubvdo.net/store/ftoducts/VDOnetAD0L- 
ive_player.asp, May 26, 1998, 5 pages. 
H. Schulzrinne et al., "Real Time Streaming Protocol 
(RTSP)", Network Working Group, Request for Comments: 
2326, Category: Standards Track, Apr. 1998, pp. 1-92. 
SG15 Plenary May 28, 1966, "Draft Recommendation 
H.323: Visual Telephone Systems and Equipment for Local 
Area Networks which Provide a Non-Guaranteed Quality of 
Service", International Telecommunication Union, 69 pages. 

* cited by examiner 



06/15/2002, EAST Version: 1.03-0002 



U.S. Patent Feb. 27, 2001 Sheet 1 of 9 US 6,195,680 Bl 




06/15/2002, EAST Version: 1.03.0002 



U.S. Patent Feb. 27, 2001 Sheet 2 of 9 US 6,195,680 Bl 




06/15/2002, EAST Version: 1.03.0002 



U.S. Patent 



Feb. 27, 2001 Sheet 3 of 9 



US 6,195,680 Bl 




06/15/2002, EAST Version: 1.03.0002 



U.S. Patent Feb. 27, 2001 Sheet 4 of 9 US 6,195,680 Bl 




06/15/2002, EAST Version: 1.03.0002 



U.S. Patent 



Feb. 27, 2001 



Sheet 5 of 9 



US 6,195,680 Bl 



Client agent sends a request 
message to the control server for 
the multimedia stream. 



The control server determines which 
set of streaming servers the client 
agent should connect to. 
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The selected streaming server starts 
to deliver the real-time multimedia 
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The client agent detects that the 
streaming server fails. 
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server to switch it to an alternate server 
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and alternate entries from the client 
agent to determine its alternate set 
of streaming server. 



The control server selects a 
streaming server among all that are 
using even-numbered ports based 
on some load-balancing heuristics. 
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The control server selects a 
streaming server among all 
that are using odd-numbered 
ports based on some 
load-balancing heuristics. 



The selected streaming server starts to 
deliver the real-time multimedia data to the 
client agent, it now becomes the primary 
streaming server. 
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CUENT-BASED DYNAMIC SWITCHING OF 
STREAMING SERVERS FOR 
FAULT-TOLERANCE AND LOAD 
BALANCING 

FIELD OF THE INVENTION 5 

This invention relates generally to providing fault toler- 
ance and load balancing for real-time data streaming. More 
particularly, it relates to a client-based dynamic server 
switching method for use in a distributed system including 10 
multiple servers that are simultaneously transmitting one or 
more real-time multimedia streams. 

BACKGROUND 

The demand for real-time multimedia streaming is 15 
steadily increasing. In order to increase the scale of network 
broadcast using real-time multimedia streaming, multiple 
servers can be used to provide the same multimedia stream 
to a large number of clients. Clients are directed to one of a 
multiplicity of servers to obtain the multimedia stream in 2 q 
real-time. 

One method known in the art that seeks to increase the 
processing capacity at hot sites on the Internet is to create a 
cluster of computing nodes (also called a multi-node cluster) 
to handle the load. The Internet refers to the collection of 25 
networks and gateways that use the Transmission Control 
Protocol/Internet Protocol (TCP/IP) suite of protocols. TCP/ 
IP is a well known protocol that was developed by the 
Department of Defense for communications between com- 
puters (see e.g., D. E. Comer, "Internetworking with TCP/IP, 30 
Principals, Protocols and Architecture /'Prentice Hall, which 
is hereby incorporated by reference herein in its entirety). 
The multi-node cluster is (encapsulated) made to appear as 
one entity to clients, so that the added capacity provided by 
the multi-node cluster is transparent to clients. Client 35 
requests are distributed among nodes in the multi-node 
cluster. Many load balancing techniques are known in the 
art; see for example, Dias et al, "A Scalable and Highly 
Available Web Server", Proc. 41st IEEE Computer Society 
Intl. Conf. (COMPCON) 1996, Technologies for the Infer- 40 
mation Superhighway, pp. 85-92, February 1996; see also 
U.S. Pat. No. 5,371,852, issued Dec. 6, 1994 to Attanasio et 
al., entitled "Method and Apparatus for Making a Cluster of 
Computers Appear as a Single Host." 

Research and development has also been increasing in 45 
both the areas of audio/video streaming and video confer- 
encing. Video conferencing differs from audio and video 
streaming in that the communication is bi-directional and 
end-to-end delays must be very low (<200 ms) for interac- 
tive communication. In fact, video conferencing standards 50 
are quite mature and have emerged from both the Interna- 
tional Telecommunication Union (ITU), in the form of the 
H.3xx standards; and the Internet Engineering Task Force 
(IETF) in conjunction with the multicast backbone 
(MBONE). In general, the two camps use the same audio 55 
and video compression standards (defined by the ITU), but 
differ in their networking protocol specifications. 

Audio/video streaming differs technically from its video 
conferencing counterpart in that it can afford greater flex- 
ibility in end-to-end delays when being transmitted across a 60 
network, and in the fact that stored content may be manipu- 
lated off-line with additional processing. These begin to 
merge when one considers live audio and video streaming 
applications (e.g., Internet, radio, and TV). The most rel- 
evant of the ITU standards is H.323, which defines audio/ 65 
visual services over LANs for which quality of service 
cannot be guaranteed (see e.g., "Draft Recommendation 
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H.323: Visual Telephone Systems and Equipment for Local 
Area Networks Which Provide A Non- Guaranteed Quality 
of Service," (May 28, 1996), which is hereby incorporated 
herein by reference in its entirety). This standard specifies a 
variety of audio and video coders and decoders (CODECs) 
as well as signaling protocols to negotiate capabilities and 
setup and manage connections. The underlying transport 
specified is the Real-time Transport Protocol (RTP). This 
protocol, defined by the IETF, is intended to provide a means 
of transporting real-time streams over Internet Protocol (IP) 
networks, A new protocol, Real Time Streaming Protocol 
(RTSP), more directly addresses the issues of delivering and 
managing multimedia streams, (see e.g., "Real Time Stream- 
ing Protocol (RTSP)" IETF Request for Comments: 2326 
(April 1998), which is hereby incorporated herein by refer- 
ence in its entirety). Clearly, this area is still evolving as new 
protocols are being defined and refined to satisfy a wide 
range of emerging networked multimedia applications. 

Streaming technology can be used to deliver live audio 
and video data, where the clips arrive in streams so that users 
can begin to view or hear the clip before the download is 
complete. Conventional Internet traffic is short-lived, with a 
duration ranging from milliseconds to seconds, and bursty. 
In contrast, real-time multimedia streaming is lengthy, with 
a duration ranging from minutes to even hours, with low 
continuous bandwidth requirements. Server and/or network 
failures will terminate the real-time streaming process, and 
the stream from a given server will be interrupted for a 
particular session. This interruption may, in many cases, 
only be detected at the client. Thus, a need exists for a 
client-based means to automatically switch to an alternate 
server in order to continue receiving a multimedia stream in 
an uninterrupted fashion in the event of a service 
degradation, load imbalance, or failure. The present inven- 
tion addresses such a need. 

One known method in the art for increasing scalability for 
real-time multimedia streaming is through the use of 
so-called "reflector*' technology. The reflector technology is 
used in applications — such as IBM's BAMBA™, Vosaic's 
MED1ASERVER™ and White Pine Software's 
CU-SeeMe™ — to provide real-time audio and video stream- 
ing over the Internet. Reflectors are servers that manage the 
distribution of audio and video streams to their receivers. 
They can be cascaded and scaled to handle increased 
demand for a broadcast. Multimedia streams are replicated 
at each reflector and delivered to multiple receivers. By 
simply adding more reflectors, a broadcast is capable of 
supporting large numbers of clients. 

A key problem with the basic reflector technology is that 
the resulting workload across reflectors is poorly balanced 
and network bandwidth can be wasted. For example, some 
reflectors can have a large number of clients simultaneously 
connecting to it while others are serving only a few clients. 
This causes the workload across reflectors to be highly 
unbalanced and can cause performance degradation due to 
server and/or network overloads. Another problem with the 
basic reflector approach, is that in the event that a reflector 
(server) fails or is overloaded, there is no automatic mecha- 
nism for redirecting a client to another server where they 
will continue receiving the multimedia stream. The present 
invention addresses these problems. 

SUMMARY 

Accordingly, it is an object of this invention to provide an 
improved client-based system for the fault- tolerant delivery 
of real-time or continuous data streams. In one embodiment, 
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real-time multimedia streams, e.g., live audio and video 
clips, are delivered in streams so that users can begin to view 
or hear the clip before the download is complete. The 
real- time multimedia streams can be lengthy, with durations 
ranging from minutes to even hours, with low continuous 5 
bandwidth requirements. According to the present invention, 
a receiver (also called a client) automatically detects load 
imbalances and/or failures (complete or partial) and dynami- 
cally switches to an alternate server in order to continue 
receiving the real-time multimedia stream with minimal 10 
disruption. 

The present invention includes features for automatically 
and gracefully switching clients among multiple servers in 
the event that a server becomes overloaded or fails. A 
preferred embodiment addresses the case where clients are 15 
receiving a continuous multimedia stream and the switching 
must be transparent to the client and maintain uninterrupted 
playback of the multimedia streams. When a server fails, its 
respective client agents detect the failure and automatically 
switch to alternate servers that continue to provide the client 20 
agents with the real-time multimedia streams. 

The present invention also includes features for gracefully 
switching client agents to alternate servers in the case of a 
server failure or overload while maintaining a balanced load 
across multiple servers in a distributed network environ- 25 
ment. 

An example of a system having features of the present 
invention includes: a control server; two or more streaming 
servers, and a plurality of client agents. The control server 
is preferably a scalable server that is capable of handling a 
requests from a large number of incoming client agents and 
redirecting them to the streaming servers that are providing 
the multimedia data. The control server assigns different 
identifiers to the streaming servers for delivering the mul- 
timedia data, and uses these identifiers to group these 
streaming servers into two or more different sets. The 
streaming servers are used to de fiver the real-time multime- 
dia streams to the client agents. To receive a multimedia 
stream, client agents are given an identifier to connect to a 
server in one of the sets. 

Each client agent receives the multimedia stream from a 
streaming server, performs the appropriate processing (e.g., 
decompression, scaling) on the stream and renders the 
multimedia output. Each client agent can be provided with 45 
a primary server identifier as well as a secondary server set 
identifier. The primary entry characterizes the primary 
streaming server in the set of servers the client agent is 
connecting to. The secondary entry characterizes the set 
containing an alternate server for the client agent. When a 5Q 
client detects a failure or overload, the client sends a switch 
request to the control server which then selects a server in 
the secondary set and redirects the client agents of the 
primary server to the selected alternate server. Thus, the 
client agents can continue to receive the multimedia streams S5 
with minimal or no interruption. 

BRIEF DESCRIPTION OF THE DRAWINGS 

These, and further characteristics and features of the 
invention, will be more apparent from the following detailed 60 
description of a preferred embodiment and the appended 
drawings in which: 

FIG. la depicts an example in block diagram form of an 
environment having features of the present invention; 

FIG. 16 depicts an example of a data structure which can 65 
be used to record the relationship between a client and the 
streaming servers depicted in FIG. la; 
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FIGS. 2(a), 2(b), and 2(c) depict an example in block 
diagram form of a client agent requesting a multimedia 
stream from the streaming servers; 

FIGS. 3(a), 3(b), and 3(c) depict an example in block 
diagram form of the dynamic server switching process for 
the client agent when the primary server for the client agent 
fails; 

FIG. 3d depicts a more detailed example of the client- 
server architecture of FIGS. 3(a)-(c), adaptable to the 
present invention; 

FIG. 4 depicts an example of a method having features of 
the present invention for streaming continuous real-time 
MM data to a client; 

FIG. 5 depicts an example of a method having features of 
the present invention for switching clients among multiple 
servers if a streaming server becomes overloaded or fails; 

FIG. 6 depicts another example of an architecture having 
features of the present invention; 

FIG. 7 depicts a more detailed example of the source, 
client and reflector depicted in FIG. 6; and 

FIG. 8 depicts an example of a hierarchical reflector 
configuration. 

DETAILED DESCRIPTION 

FIG. la illustrates an example of an environment includ- 
ing a server architecture 1.7 and a client agent 1.8 having 
features of the present invention. The server architecture 1.8 
includes a control server 1.1 and at least two sets (1.5, 1.6 
) (also called clusters) of streaming servers (1.2, 1.3). Each 
set of streaming servers includes at least one streaming 
server (1.2, 1.3), each having a number of ports (1.92, 1.93). 

In one embodiment, the streaming servers (1.2, 1.3) are 
assigned to one of the sets (1.5, 1.6) based on a simple 
arithmetic test. For instance, the assignments may be based 
on connection port numbers, e.g., all streaming servers using 
even-numbered ports 1.92 are assigned to one set, and all 
streaming servers (1.2, 1.3) using odd-numbered ports 1.93 
are assigned to another set. The server set assignments could 
further include considerations including one or more of: 
size; capacity; location (locality/affinity); and network 
connectivity, so that a good balance of server availability can 
be maintained in the two sets. One method to achieve this is 
for the control server 1.1 to assign any new streaming server 
to the currently smaller set. Servers in the system are 
preferably divided into two disjoint non-empty sets — a set 
which is serving stream data through the odd-numbered port 
and another set which is serving stream data through the 
even-numbered port. Both of these sets (1.5, 1.6) contain at 
least one server (1.2, 1.3). The two sets of servers are 
preferably mutually exclusive, meaning that no servers in 
the system can be in these two groups at the same time. 

Those skilled in the art will appreciate that many alter- 
natives are available. The control server 1,1 could be a 
gateway through which client requests must pass and which 
includes a routing function to distribute client requests 
among servers in the cluster. For example, a system adapt- 
able to the present invention that provides load balancing in 
an encapsulated cluster of nodes is described in U.S. Pat. No. 
5,371,852, issued Dec. 6, 1994, entitled "Method and Appa- 
ratus for Making a Cluster of Computers Appear as a Single 
Host", by Attanasio et al ('852 patent). The present invention 
has a common assignee with this U.S. patent, which is 
hereby incorporated by reference in its entirety. Here, only 
the address of a Transmission Control Protocol (TCP) router 
(control server 1.1) is given out to clients 1.8; the TCP router 
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distributes incoming requests among the nodes (streaming 
servers 1.2, 1.3) in the cluster, either in a round -robin 
manner, or based on the load on the nodes. In the J 852 
patent, a Virtual Encapsulated Cluster routes TCP informa- 
tion that crosses the boundary of a computer cluster. The 5 
information is in the form of port type messages. Incoming 
messages are routed and the servers respond so that each 
cluster appears as a single computer image to the external 
host. A cluster of servers with a single TCP-router node is 
divided into a number of virtual clusters (virtual encapsu- 10 
lated clusters). Each virtual encapsulated cluster appears as 
a single host to hosts on the network which are outside the 
cluster. The messages are routed to members of each virtual 
encapsulated cluster in a way that keeps the load balanced 
among the set of cluster nodes. The TCP router can also act 15 
as a proxy, where the requests are sent to a selected node, 
and the responses go back to the TCP router and then to the 
client. In a preferred mode of operation, called forwarding 
mode, client requests are sent to a selected node, and the 
responses are sent back to the client directly from the 20 
selected node, bypassing the router. 

Referring again to FIG. la, as is conventional, the client 
1.8 can be directly connected to the control server 1.1 or 
over a network (not shown) such as a cable network, 
telephone network, local area network (LAN), an intranet, or 25 
the Internet (or some combination of these). The client agent 
1.8 (also called simply client), can be any conventional 
computer or processor-based machine with a processor, 
memory and operating system and application software and 
networking (hardware and software) to communicate 30 
requests and receive data streams from a streaming server. 

The control server 1.1 redirects incoming client agent 1.8 
requests to the streaming servers (1.2, 1.3), preferably while 
monitoring the workload of the streaming server. The con- 
trol server is preferably a scalable server that is capable of 35 
handling a large number of incoming client agent requests 
and redirecting them to the streaming servers that are 
providing the multimedia data. The control server assigns 
different identifiers to the streaming servers for delivering 
the multimedia data, and uses these identifiers to group the 40 
streaming servers into two or more different sets (1.5, 1.6). 
By way of example only, one set 1.5 of streaming servers 1.2 
is delivering multimedia streams through even- numbered 
ports 1.92 and another set 1.6 of streaming servers 1.3 is 
delivering the multimedia streams through odd-numbered 45 
ports 1.93. Port -based routing of TCP connections is well 
known in the art (see for example the aforementioned '852 
patent). 

As is also conventional, communication channels 1.4 5Q 
exist between the control server 1.1 and the streaming 
servers (1.2, 13), allowing the control server to preferably 
monitor the workload of the streaming servers while redi- 
recting incoming client agent 1.8 requests to them. 

Each instance of the streaming process begins with a 55 
client agent 1.8 connecting to the control server 1.1 request- 
ing the multimedia stream. The control server then assigns 
and redirects the client to one of the streaming servers in 
either of the two groups (1.5, 1.6). The assignment can be 
based on a conventional round-robin approach or based on 60 
some load-balancing heuristics. For example, the server 1.1 
can redirect the client 1.8 to a streaming server based on a 
weighted round-robin approach, or to a streaming server 
having a lowest utilization rate. 

An example of a system for weighted TCP routing is 65 
described in co-pending U.S. patent application Ser. No. 
08/701,939, filed Aug. 23, 1996, now U.S. Pat. No. 5,918, 



,680 Bl 

6 

017, entitled "Weighted TCP Routing to Service Nodes in a 
Virtual Encapsulated Cluster" by C. Attanasio, G. Hunt, G. 
Goldszmidt, and S. Smith (IBM Docket No. Y0996167), 
now U.S. Pat. No. 5,918,017. The present invention has a 
common assignee with this co-pending patent application, 
which is hereby incorporated herein by reference in its 
entirety. 

An example of a F ault Tolerant Recoverable TCP/I P 
Connection pouter (KIR-CK) and methods nt connecting at 
least two FTR-CRs to multiple systems, where the FTR-CRs 
have synchronized internal tables and are capable of switch- 
ing between active and standby states is described in 
co-pending U.S. patent application Ser. No. 08/929,409, 
entitled "FAULT TOLERANT RECOVERABLE TCP/IP 
CONNECTION ROUTER," by Baskey et al, filed Sept. 15, 
1997, (IBM Docket No. Y0997232). The present invention 
has a common assignee with this co-pending patent 
application, which is hereby incorporated herein by refer- 
ence in its entirety. A recoverable TCP/IP Connection Router 
is a virtual encapsulated cluster which has two TCP-router 
nodes, a primary and a backup. The server cluster is aug- 
mented with a recovery manager which causes the backup 
TCP-router to become active if the primary fails. The 
connection state at the time of failure can be reconstructed 
by (or alternatively known at) the backup router so that zero 
or a minimum number of client connections will be lost due 
to failure of the TCP-router node. The configuration/ 
management information of the virtual encapsulated cluster 
is also replicated (or constructed) at the backup. Finally, the 
start up protocol of the TCP-router node is changed so that 
recovery of the primary router will not cause a failure in a 
backup that has taken over for it. 

An example of an affinity-based router and method for 
routing and load balancing in an encapsulated cluster of 
server nodes is described in co-pending U.S. patent appli- 
cation Ser. No. 08/947,361, filed Oct. 8, 1997, entitled 
"Affinity-based Router and Routing Method," by Dias et al., 
(IBM Docket No. Y0996265). The present invention has a 
common assignee with this co-pending patent application, 
which is hereby incorporated herein by reference in its 
entirety. The affinity-based system includes a multi-node 
server, wherein any of the server nodes can handle a client 
request, but wherein clients have affinity to one or more of 
the server nodes that are preferred to handle a client request. 
Such affinity is due to state at the servers either due to 
previous routing requests, or data affinity at the server. At the 
multi-node server, a node may be designated as a TCP 
router. The address of the TCP router is given out to clients, 
and client requests arc sent thereto. The TCP router selects 
one of the nodes in the multi-node server to process the 
client request and routes the request to this server; in 
addition, the TCP router maintains affinity tables, containing 
affinity records, indicating which node a client was routed to. 
In processing the client request, the server nodes may 
determine that another node is better suited to handle the 
client request, and may reset a corresponding TCP router 
affinity table entry. The server nodes may also create, modify 
or delete affinity records in the TCP router affinity table. 
Subsequent requests from this client are routed to server 
nodes based on any affinity records, possibly combined with 
other information (such as load). 

An example of a port-based TCP connection router which 
includes load balancing and high availabilty functionality is 
available from the IBM Corporation under the trademark 
"ENETWORK DISPATCHER." Those skilled in the art will 
appreciate however that any connection routing or load 
balancing heuristics known in the art can be applied to the 
system described herein. 
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Each client agent 1.8 can be provided with a primary and 
a secondary server identifier. The primary entry character- 
izes the set containing the streaming server the client agent 
is initially connecting to. The secondary entry characterizes 
the set containing an alternate server of the client agent. The 5 
selected (primary) streaming server then starts the transmis- 
sion of the multimedia stream to the client agent 1.8. 
According to the present invention, the streaming servers 
(1.5, 1.6) '-an up dy namically switched in response to a cl ient 
1 .8 detectir fl a ^trfi nmT ' n ft- fi * rv rrr Fa ^ irf> 1Q 

FIG. lb depicts an example of a data structure 1.9 which 
can be used to record the relationship between the client 1.8 
and the streaming servers. As depicted, the structure 1.9 
includes a client ID 1.91 which identifies a client 1.8; a 
primary ID 1.94 which identifies a primary server 1.2 for 15 
serving the multimedia stream to the client 1.8; and a 
secondary ID 1.96 which identifies a secondary set of 
servers 1.6 which includes a secondary server 1.3 capable of 
serving this client in the event a failure is detected in the 
stream or the primary server 1.2. In a preferred embodiment, 2 q 
the data structure is maintained at the clients, to reduce the 
overhead on the control server 1.1. 

To illustrate an example of how the client-based dynamic 
switching mechanism works, assume without loss of gen- 
erality that the client agent 1.8 is assigned a primary 25 
streaming server 1.2 that is using an even-numbered port 
1.92. The primary ID 1.94 and secondary ID 1.96 could be 
communicated to the client 1.8 and stored in the data 
structure 1.9 at the client. When the client 1.8 detects a 
failure in the stream or the primary streaming server, the 3u 
client agent communicates again, for example passing the 
primary ID 1.92 and/or the secondary ID 1.96 to the control 
server 1.1, requesting (based on the secondary ID 1.96) an 
alternate streaming server 13 that is using an odd-numbered 
port 1.93. The control server 1.1 then switches the client 35 
agent to an alternate streaming server 1.3 in the set 1.6 of 
streaming servers using odd-numbered ports 1.93. The 
selected secondary streaming server now becomes the pri- 
mary streaming server for the client agent 1.8, and the set 1.5 
that contains streaming servers using even-numbered ports 40 
becomes the set for the secondary server. The control server 
1.1 preferably communicates the updated identifiers (1.94, 
1.96) to the client 1.8. This feature allows the client to 
dynamically switch back and forth between the two sets. 

In one embodiment, the multimedia stream is broadcast 45 
over the Internet and the client 1.8 is a conventional com- 
puter workstation equipped with a standard browser having 
an appropriate plug-in for the streamed video file format. 
Alternately, the browser is JAVA (a trademark of SUN 
MicroSystems) compatible, or uses other platform-specific 50 
technology, such as ACTIVEX (a trademark of Microsoft), 
so that a plug-in is not needed. 

Given this streaming server organization, the client agent 
1.8 can be guaranteed an alternate streaming server that is 
not the same as the primary streaming server during the 55 
switching process. As described above, the streaming serv- 
ers are divided into two disjoint sets, say A and B. Without 
loss of generality, assume that streaming servers in set A are 
serving the data from even-numbered ports 1.92 and those in 
set B from odd-numbered ports 1.93, respectively. 60 
Furthermore, assume that a particular client agent 1.8 is 
initially assigned to a streaming server a in set A. When 
server a fails, the client 1.8 will request the control server 1.1 
for a new (alternate) streaming server using an odd- 
numbered port 1. 93, and the control server 1. 1 will assign 65 
it a streaming server b that is in the set B. However, given 
that the two sets are disjoint, it follows that the alternate 
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streaming server must be different from the primary stream- 
ing server. The converse is true, if the client is initially 
connected to a streaming server b' in set B but requests to be 
switched to an alternate streaming server, it will be recon- 
nected to a streaming server a' that is guaranteed to be 
different from b'. 

Those skilled in the art will appreciate that the client- 
based method and system of the present invention can also 
be applied to a single set or cluster of servers wherein the 
control server 1.1 and streaming servers (1.2, 1.3) are 
appropriately configured. For example, the control server 
1.1 could be coupled to a single set of streaming servers 1.5 
which includes two steaming servers (1.2 and 1.3). One 
streaming server 1.2 could be using an even port 1.92 and 
the other streaming server 1.3 could be using an odd port 
1.93. The control server assigns different identifiers to the 
streaming servers for delivering the multimedia data. Each 
of the streaming servers uses the identifier for transmitting 
the real-time multimedia streams. Each client agent receives 
the multimedia stream from a streaming server, performs the 
appropriate processing (e.g., decompression, scaling) on the 
stream and renders the multimedia output. Each client agent 
1.8 can be provided with a primary and an alternate server 
identifier. The primary entry characterizes the streaming 
server the client agent is initially assigned and connecting to. 
The secondary entry characterizes the alternate server of the 
client agent. When the client 1.8 detects a streaming server 
failure or overload, the control server 1.1 redirects the client 
to the alternate streaming server. Thus, the client 1.8 can 
continue to receive the multimedia streams with minimal or 
no interruption and be guaranteed an alternate streaming 
server that is not the same as the primary streaming server 
as a result of the switching process. 

FIGS. 2(a)~ic) depict an example of a process in a system 
wherein a client agent can request a multimedia stream from 
the streaming servers. As depicted, the system includes a 
server architecture 2.8, and a client 2.5. The server 2.8 
includes a control server 2.1 and a set 2.6 of streaming 
servers 2.2 that are using even-numbered ports 2.92 and a set 
2.7 of streaming servers 2.3 that are using odd-numbered 
ports 2.93. The control server and the streaming servers 
communicate via communication channels 2.4. 

In FIG. 2(a), the client agent 2.5 requests a multimedia 
stream by establishing a connection 2.9 to the control server 
2.1 managing the multimedia stream. The control server 2.1 
then assigns and redirects the Client Agent's request to one 
of the sets (2.6, 2.7) of streaming servers and also to a 
particular (primary) server within a set, based on some 
load-balancing heuristics. For example, the control server 
may redirect the request to a streaming server based on the 
current number of connection streams to each streaming 
server. The control server 2.1 also assigns an alternate server 
to the requesting Client 2.5. 

For example, as depicted in FIG. 2(b), the control server 
2.1 has redirected the client agent 2.5 to the streaming server 
indicated by the link 210. As depicted in FIG. 2(c), the 
selected streaming server (2.10, 2.2) then begins delivering 
the multimedia stream to the client agent 2.5, over the link 
2.11. The control server 2.1 can provide a primary and 
alternate streaming server identifier 1.96 to the requesting 
client 2.5 or the server 2.1 can maintain that information at 
the server 2.1. 

FIGS. 3(a)-(c) depict an example of a client-based 
dynamic streaming server switching process in accordance 
with the present invention. As depicted, the system includes 
a server architecture 3.8 and a client agent 3.5. The server 
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architecture includes a control server 3.1 and a plurality of switching clients among multiple streaming servers in the 

streaming servers (3.2, 3.3). One set 3.6 of servers 3.2 are event that a streaming server (SS) (3 2, 3.3, FIG. 3) becomes 

using even-numbered ports 3.92 and another set 3.7 of overloaded or fails. In this example, the client 3.5 is receiv- 

servers 3.3 are using odd-numbered ports 3.93. Communi- ing a continuous multimedia stream and the switching must 

cation channels 3.4 link the control server 3.1 and the 5 be transparent to the client and maintain uninterrupted 

streaming servers. playback of the multimedia streams. As depicted, in step 5.1, 

In FIG. 3(a), an original connection 3.9, assigned by the the client 3.5 detects a failure. By way of example only, the 

control server 3.1 to deliver the multimedia stream from a determination can be made based on: the received bit or 

primary streaming server 3.2 to the client agent 3.5 fails. The frame rate (for video); a bit rate or sample rate (for audio); 

failure may be detected by using any known method. For 1Q monitoring the delivery rate and/or for packets arriving out 

example, the client agent 3.5 may monitor the effective bit G f order: for example using packet numbering mechanisms 

rate of the stream. If the bit rate falls below a given available in TCP; sequence numbering or time stamp capa- 

threshold, the connection to the server 3.2 can be considered bilities of RTP (in combination with the User Datagram 

to have failed. As a result of this failure, the client agent 3.5 p rot0 col (UDP)). For example, RTP includes a field allow- 

can send a message 310 to the control server 3.1, requesting m ^ uge of ^ & of nce nurnbers . In any casC) 

to be switched to an alternate server Referring now to FIG ^ determinalion coul / be 5ased on me rate meaS urement or 

3(d), the control server 3.1 then redirects 3.1 1 the Client . , . c „. . , / j- \ 

a *» i** -3* * ** n — * ii monitoring mechanism falling below (or exceeding) some 

Agent s multimedia stream request to an alternate server 3.3 . . ,j ^ i iL • >% ^ , i 

hf the set of streaming servers 3.7 that is using odd- * resho J<*- Alternatively, the primary SS 3.2 or the con rol 

numbered ports 3.93, in response to the client's switch s u erve t r . 31 c ^ d an f^at dl f ess °J switch signal to 

request. Referring now to FIG. 3(c), the selected alternate 20 the client. An explicit signal can be used for example to 

server 3.7 starts streaming data to the client agent over the switch clients * P hases with minimal disruption. An addi- 

Link 3.12 with minimal or no disruption. tional alternative is to provide users with an explicit way to 

Apreferred embodiment of the present invention includes send a switch ret l uest to the control server based on their 

features implemented as software tangibly embodied on a subjective evaluation of the output. This could be accom- 

computer program product or program storage device for 25 Pushed by one skilled in the art by equipping the client's 

execution on a processor (not shown) provided with the user interface (for example the browser or remote control) 

client (1.8, 2.5, 3.5), the control server (1.1, 2.1, 3.1) and/or with a button (or other graphical or speech driven means) 

the streaming servers (1.2, 13, 2.2, 2.3, 3.2, 3.3). For that would initiate a switch request. In step 5.2, when a 

example, software implemented in a popular object-oriented failure, distress or switch signal is detected by the client 3.5, 

computer executable code such as JAVAprovides port ability 30 the client communicates a request to the control server to 

across different platforms. Those skilled in the art will automatically switch to the alternate server. In step 53, the 

appreciate that other procedure-oriented or object-oriented control server 3.1 uses the primary and alternate entries for 

(00) programming environments, including but not limited the client agent to determine the appropriate alternate set of 

to C* and ACTIVEX, can also be employed. streaming servers. The primary 1.94 and secondary identi- 

Those skilled in the art will also appreciate that the 35 fiers 1.96 could be provided by the client (assuming they 

software may be embodied on a magnetic, electrical, optical, were originally communicated to the client) to the control 

or other persistent program and/or data storage device, server with the switch request. Alternatively, the control 

including but not limited to: magnetic disks, DASD, bubble server could manage the identifiers 1.9 for its clients. In step 

memory; tape; optical disks such as CD-ROMs; and other 5.4, the control server identifies the servers associated with 

persistent (also called nonvolatile) storage devices such as 40 the alternative set for this client (for example those using 

core, ROM, PROM, flash memory, or battery backed RAM. even-numbered ports). In steps 5.5 and 5.6 the control server 

Those skilled in the art will appreciate that within the spirit selects a server from the set of alternate streaming servers, 

and scope of the present invention, one or more of the e.g., based on some load balancing heuristics. Instep 5.7, the 

components instantiated in the memory of the clients or selected alternate streaming server 3.7 begins to provide the 

servers could be accessed and maintained directly via disk 45 client with the real-time multimedia streams and the alter- 

(not shown), the network, another server, or could be dis- nate SS becomes the new primary SS. The original (or 

tributed across a plurality of servers. another) streaming server is selected as the new alternate SS 

FIG. 4 depicts an example of a method having features of for the client, 

the present invention for streaming continuous real-time Those skilled in the art will appreciate that the client could 

data streams to a client. As depicted, (with reference to FIG. 50 be enhanced within the spirit and scope of the present 

2) in step 4.1, a client 1.1 sends a request for a multimedia invention to further assist in the secondary server selection 

(MM) stream to the control server. The streaming servers process. For example, when a client 1.1 sends a request for 

(SS) (2.2, 2.3) have been divided into at least two non- a multimedia (MM) stream to the control server, the server 

overlapping sets where each set includes at least one SS. In could download code (for example by means of an applet) 

step 4.2, the control server identifies a set of primary 2.6 55 including stream or system configuration dependent rules 

(and alternate 2.7) streaming servers that the client 2.5 can which would enhance the client's selection process based on 

connect to. In steps 43-4.5, depending on the set selected as the nature of the failure. The client could for example 

primary, the control server selects a primary SS from the set. determine a delivery rate based on the difference between a 

The control server can optionally select the SS 2.2 from a server's time stamp and the time a packet is delivered to the 

plurality of SSs in the selected set 2.6 using conventional eo client. If the delivery rate falls below a threshold this could 

load-balancing techniques. In step 4.6, the selected SS be determined to be a network failure (or server congestion) 

begins streaming 2.11 a continuous real-time MM data to the for which the downloaded rules indicate a certain secondary 

client 2.5, In this example, the control server 2.1 also server set preference. 

communicates 2.12 an identifier of the primary and alternate Those skilled in the art will appreciate that although a 

streaming servers to the client 2.5. 65 preferred embodiment is described in terms of the TCP/IP 

FIG. 5 depicts an example of a method having features of suite of protocols, the present invention can be applied to 

the present invention for automatically and gracefully any packet-based data delivery system. A packet refers to a 
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unit of information transmitted as a whole from one device An example of a media-streaming system — adaptable to 

to another on a network. In packet switching networks, a the present invention — and capable of delivering audio and 

packet typically has a fixed maximum size and includes both video over low-bandwidth modem connections using stan- 

data and a header including an identification number, source dard compression technologies is available from the IBM 

and destination addresses and sometimes error control data. 5 Corporation under the trademark BAMBA. BAMBA is 

Information is disassembled into packets before routing. currently available from the IBM Corporation. BAMBA 

Each packet may travel along a different route and arrive at offei ? high-quality audio and video over low-bit-rate con- 

their destination at different times or out of sequence and the ° cctl0 r ns ^ d can , °P cratc „ USin S a standard Hv P e ^ Tc * 

receiving computer reassembles the original information. Transfer Protocol (HTTP) server. A BAMBA video is 

Standards for packet switching on networks are documented 10 enhanced ™ th ***** Prions for reducing the effect of 

in CCIT recommendation X 25 errors in a lossy-network environment. BAMBA adheres to 

existing standards wherever possible. BAMBA enables the 

Streaming Data on the Internet and World Wide streaming of audio and video across the Web through 

Web very-low-bit-rate connections; audio is sufficiently com- 

- pressed to stream over 14.4 Kb/s- and video over 28.8 Kb/s- 

Today's computer-network infrastructures, including the modem connections. 
Internet, were not designed with data streaming in mind. BAMBA not only achieves the low-bit-rate goal, but can 
Streaming media requires that data be transmitted from a also be cxt e D ded to support higher-bit-rate streams to pro- 
server to a client at a sustained bit-rate, high enough to vidc higher-quality streaming over intranets or higher- 
maintain continuous and smooth playback at the receiving ^ bandwidth Internet connections. Furthermore, when stream- 
chent-station. m g jg no ^ possible, because of congestion or insufficient 

The World Wide Web (WWW) has become a primary bandwidth availability, the BAMBA player (client software) 

means of disseminating information. Initially, the type of at the receiving client automatically calculates how much 

information distributed was mostly in the form of text and data to pre-load in order to maintain continuous playback, 

graphics. Later, images and stored audio and video files ^ This allows clients connected via low-bit-rate connections to 

emerged. These audio and video files are downloaded from fall back to a download-and-play mode and still receive the 

a server and stored at the client before they are played. More higher-bit-rate content. 

recently, streamed audio and video has become available The present invention provides a mechanism which 
from both stored and live sources on the Web. Audio and enables a client agent of streaming applications to continue 
video streaming enables clients to select and receive audio 30 receiving data from an alternate server with minimal 
and video content from servers across the network and to disruption, for example if the primary server fails. A con- 
begin hearing and seeing the content as soon as the first few ventional scalable server architecture such as a router 
bytes of the stream arrive at the client. Streaming technology capable of routing and preferably load balancing requests 
involves a combination of audio and video compression, among a set (or "cluster") of TCP-based servers can be used 
stream formatting and transmission packetization schemes, 35 m 0 ne embodiment of the present invention. This mecha- 
networking protocols and routing, client designs for display- D is m is preferably fault-tolerant and suitable for large-scale 
ing and synchronizing different media streams, and server audio and/or video distribution applications where multiple 
designs for content storage and delivery. servers are available for delivering the streaming data. One 

An example of a method and apparatus for transmission immediate application is to implement this dynamic client- 

of video image data over an internet network, which can 40 based server switching scheme in a live broadcast applica- 

include an ordinary telephone network, is described in tion that uses TCP connection router logic as part of the 

co-pending US patent application Ser. No. 08/697,652 now control server 1.1 together with five audio and video stream- 

U.S. Pat. No. 5,793,895, entitled "Intelligent Error Resilient ing technology. An example of this streaming technology is 

Video Encoder" (IBM Docket No. YO 996- 165), by Y BAMBA, which includes streaming audio and video players 

Chang, Z.-Y. Sbae and M. Willebeek-LeMair ("Chang et 45 designed to provide high quality audio and high frame-rate 

al./'). video from a live source over low-bandwidth connections. 

The Chang etal., patent application is commonly assigned FIG. 3d depicts a more detailed example of the client- 

with the present invention and is hereby incorporated by server architecture of FIG. 3a adaptable to the present 

reference herein in its entirety. Chang et al., increases the invention. As depicted, the system includes a client 350' and 

error resilience of a compressed data stream that is trans- 50 a server 320'. The server 320' includes standard HTTP Web 

mitted over a low bit rate network and provides for fast server logic 321, and one or more multimedia streaming 

recovery from packet losses in a congested network envi- files, such as BAMBA audio and video files, stored in 

ronment. In particular, Chang et al, provides for a method memory 322. The client 350' is a conventional computer 

of compressing data to improve its tolerance to error. The which includes a standard Web browser (such as is available 

actual video encoders can use H.263, H.261, or other well 55 from Netscape under the trademark NAVIGATOR or from 

known video compression algorithms. Data is divided into a Microsoft under the trademark INTERNET EXPLORER) 

number of frames, where each of the frames is divided into and the BAMBA audio and video plug-in software 352. 

macroblocks. Each macroblock is then assigned a depen- The BAMBA plug-ins 352 are implemented as a set of 

dency count, where the count is a measure of the depen- conventional dynamic link libraries (DLLs) that interface to 

dency of macroblocks in subsequent frames. If the depen- so the Web browser 354 through the Netscape-defined plug-in 

dency count of the blocks exceeds a selected threshold, then API (NSP Interface 353). Those skilled in the art will 

intra -coding of the block is applied. If the dependency count appreciate that Netscape has defined a set of plug-in routines 

of the block does not exceed the threshold, then inter-coding which are used to communicate between the plug-in 352 and 

is applied to the block. With intra-coding, the data in the the browser 354. Each plug-in library contains an initializa- 

block is compressed with no dependence on any other 65 tion routine within which is identified the Netscape plug-in 

blocks, while with inter-coding, the macroblock is com- routines to be used by the plug- in. Triese routines include 

pressed as a difference to other blocks. mechanisms to create and delete instances of a plug-in, 
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manage the plug-in display window, control the flow of data Hence, it is rarely possible to guarantee performance in this 
streams to the plug-in, etc. In general, the plug-in is tightly "best-effort" environment, where processing and bandwidth 
integrated with the browser Note that while Netscape was resources are typically evenly distributed among all corn- 
used in this example, the approach is similar for other peting applications. Consequently, when an audio/video clip 
browsers. 5 k accessed over the network, there is no guarantee that the 

. . . iU r, AliXTtA fi1 „ Ua ~ m u*AA*A resources (bandwidth and processing) are available to render 

As is conventional, the BAMBA files may be embedded v ° J hawda k«„ 

. TT /-p t ~\ & i r „ a ruT\AT\ n „„ ar .u*, mo * n mf the clip smoothly. To handle this situation, BAMBA has a 
in HyperText Markup Language (HTML) pages by means of . tU * a • « i . .u , 

rmr ■ £i Q * uttd i^i JT, ^«^r built-in rate monitor that dynamically evaluates the effective 

a URL pointing to file on the HTTP 321 or video server. , . J . . Y , . , r A 

When the URL is requested, the server 320- passes the ^-transfer rate (r) of a selected audroMdeo chp and 
metadata identifying the BAMBA file and containing infor- 10 compares this to the specified bit-rate , («b) for the chp, 

u ♦ #u ei «rv t-ko ai^ *™ which is contained m the chp header. If the specified rate is 

mation about the file type to the client 350'. The file type is , . . , 4 * , r , , . 

« i ii * „ . , . # . „ • . ^„„ . less than the measured rate, the chp can be rendered lmme- 

used by the browser 354 to launch the appropriate plug-m . « « . « j , t . 

1M / , . l4 , r, AUDAC1 diately. If, on the other hand, the specified rate exceeds the 
352 to playback the BAMBA file. / . \ c J r*i_ i- • i_ «? j 

r J . . # measured rate (a+b>r), then a fraction of the clip is buffered 

By way of example only, it is as^med that video and/or sufficient for me clip to play to completion smoothly once 

audio clips are streamed from standard HTTP Web servers rendering ^ startecL ^ amount of p re -buffering is d«L 

321 without special streaming software on the server and as (1 _ r /( a+b )), wher e L is the chp length. This calculation is 

can use the standard communication mechanisms provided performe <i based on the initial download rate and again 

by the HTTP protocol. This approach has certain anytime the buffer underflows. 

advantages the greatest of which is that it is simple and Rg qow to RGS ^ ^ tQ ^ 

mapsgraceiMymtomee^^ J ^ p hig-in3S2 includes logic for the 

As aresult, content creators can easily Produce audio and cHent 350 . t0 detect that the streaming server 320' has failed, 

video clips and embed them in standard HTMLpages, which and ^ ^ communicale a switch x to me control 

are then loaded onto and accessed from a standard HTTP ^ 31 ^ automalicall switch lhe client 350 , t0 

™ 321 ™» *V u "fu P a? US a ? 25 server in an alternate set of servers 3.7 available to the client. 

HTTP is TCP/IP, which provides reliable end-to-end ^net- of e ^ determination can be made 

work connections, no special provisions are required for ^ ^ ^ Wt Qr frame ^ (fa yideo) a bit fate 

handling packe loss within the network In essence the orsamplerate(foraudio); monitoring the deUvery rate or for 

a^orvideochp^ packets arriving out of order: for example using packet 

HTTP object such as an HTML or GIF format file If ^ urnberi mec l anisms available in TCP ; sequence num- 

S^Ai? e ChP * traasfe 5? u d V t tTu™™* ^ Bering or time stamp capabilities of RTP (in combination 

as TCP^P can move it and the chent 350' begins decoding ^ * e User ^ (UDp)) ^ {& RTp 

and rendering the file as soon as the first few bytes arrive. a ^ ^ ^ of ^ Qr s£quence 
Since TCP/IP is used as the underlying communication numbers . In any case> the determination could be based on 
protocol, the streams can traverse firewalls without any 35 me effect i ve rate measurement or monitoring mechanism 
special configuration requirements. In general, systems that faUing below (or exceeding) some threshold. Alternately, the 
are based on UDP/IP cannot traverse firewalls without pr i mary ser ver 320' or the control server 3.1 could send an 
explicit permission changes in the firewall which would explicit or switch signal t0 ^ client 350 , ^ 
aUow passage to the UDP/IP packets. This is because explicit signa] can be ^ for exampie t0 switch c]ients ^ 
UDP/IP packets are easier to imitate than TCP/IP packets, ^ phases with minimal disruption. In this case, when a failure, 
since the UDP/IP protocol involves no end-to-end hand- distress Qr switch signal ^ detected by the client 350^ the 
shakes or sequence numbers (see e.g., W. R. Cheswick and dient communicates a request to the control server to 
S. M. Bellovin, "Firewalls and Internet Security," Addison- automatica ii y swi tch to the alternate server. The control 
Wesley, 1994). server 3.1 can use the primary and alternate entries from the 
The audio and video technology used can be based on 45 cuen t 350' to determine an appropriate alternate set of 
standard algorithms originally defined within the ITU H.324 streaming servers. The primary and secondary identifiers 
standard for video telephony over regular phone lines (see could be provided by the client 350' (assuming they were 
e.g., "Terminal for Low Bitrate Multimedia Communica- originally communicated to the client) to the control server 
tions" ITU-T Recommendation H.324, 1995). The audio 31 with the switch request. Alternatively, the control server 
standard, G.723, specifies two bit rates: 5.3 Kbps and 6.3 50 3. 1 could manage the identifiers for its clients 350'. Next, the 
Kb/s (see e.g., "Dual Rate Speech Coder for Multimedia control server 3.1 identifies the servers associated with the 
Communications Transmitting at 5.3 and 6.3 Kbit/s", ITU-T alternative set for this client (for example those using 
Recommendation G723, 1996). even-numbered ports) and selects a server from the set of 
Returning again to FIG. 3d, a simple framing technique alternate streaming servers, e.g., based on some load bal- 
for smooth playback can be implemented wherein the audio 55 ancing heuristics. The selected alternate streaming server 3.7 
and video are interleaved into a single file to simplify the then begins to provide the client with the real-time multi- 
server 320 1 function. Essentially, the server can treat the media streams. Finally, the alternate server becomes the new 
multimedia file as any other data file. The audio and video primary server and the original server 320' (or another) 
data is interleaved proportionately to maintain a synchro- streaming server is selected as the new alternate server for 
nous rendering of both streams 355 at the client 350'. For so the client 350'. 

example, the frames can consist of a 240-byte segment of FIG. 6 depicts another example of an architecture having 

audio and a 240 b/a-byte segment of video, where a is the features of the present invention. As depicted, this sample 

audio rate and b is the video rate. application includes: a manager 640 with a controller 641 

When the Web is accessed, the actual connection speed and dispatcher 642 subsystem; a plurality of reflectors 610 
between a client 350' and a server 320' in the network varies, 65 grouped into primary and secondary (tertiary, etc.) sets 

depending on the access method (e.g., modem or LAN), the Rl . . . Rn; one or more client playback stations 620; one or 

network load, the server load, and even the client load. more multimedia source capture stations 630; and a control 
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server 640. The manager component 640 configures the 
system. Within the manager, the controller 641 assigns 
reflectors 610 to sets (Rl, R2, . . . Rn) and sources 630 to 
reflectors. The dispatcher 642 assigns primary and second- 
ary reflectors to a client based on the desired source they 5 
request to receive and on the system load characteristics. A 
multimedia source capture station S2 generates the audio 
and video data and delivers the stream to the reflectors 610. 
The reflectors 610 receive the audio and video data, and 
provide scalability by replicating the stream and delivering Q 
it to the client playback station CI. The client playback 
station CI receives the stream and decodes the audio/video 
for the human user. 

FIG. 7 depicts a more detailed example of the source, 
client and reflector depicted in FIG. 6. As depicted, the 15 
source (also called "capture") station 730, audio and video 
inputs are captured/converted from analog to digital form, 
compressed, and then packetized using conventional tech- 
niques. The packets are transmitted to the reflector 710 via 
a TCP/IP connection 735 that is established between the 20 
reflector 710 and the capture station 730. The reflector 710 
then establishes and manages one or more connections 745 
to interested clients (also called "playback stations"). These 
connections are initiated by the playback stations 720, which 
can either establish a direct connection to the reflector given ^ 
the correct IP address, or establish a connection indirectly 
via a Web server (not shown) through a HTTP URL, which 
returns a file to the playback station 720 containing the 
appropriate address information and MIME type. The reflec- 
tor 710 distributes the audio and video streams to the various 30 
playback stations 720 in the network. Playback stations may 
join an ongoing session (live broadcast in progress) at any 
point in the transmission. The reflector preferably maintains 
a circular buffer queue 712 containing the most recent 
several seconds of a live transmission for each playback 35 
station 720 to which it is connected. When a new station 
connects, the reflector produces a new copy of the circular 
buffer queue 712 for that connection. Each of the circular 
buffers is written to by the incoming capture station input 
735 and read from by the TCP/IP connection 745 to the 40 
corresponding playback station. The TCP/IP approach 
allows the connections to easily traverse firewalls and main- 
tain high quality. 

Those skilled in the art will appreciate that the same 
physical reflector node can be used for multiple sessions. 45 
The reflector node resource is limited, but upper bounds can 
be set for the number of connections per session as well as 
for the total number of connections per reflector. 
Furthermore, the reflectors may be cascaded to scale and 
handle increased demand. A reflector may also be configured 50 
to provide multicast services when it is connected to net- 
works with multicast capability (for example a multicast 
backbone (MB ONE)). For example, point-to-point TCP/IP 
connections may be established between reflectors through 
firewall boundaries that separate intranets from the global 55 
Internet. Within the intranets, the reflector may establish 
multicast UDP/IP connections to local playback stations. 

FIG. 8 depicts an example of a hierarchical reflector 
configuration. As depicted, a capture station 811 in intranet 
A 810 is transmitting to a 'root* reflector 853 in the Internet eo 
850, which in turn is forwarding the signal to reflectors 
within different intranets 810-830. Within each intranet, the 
signal is multicast to local playback stations 880. Modifi- 
cations to the streams could be made locally at each reflec- 
tor. 65 

Referring again to FIG. 6, the control server 640 is 
responsible for handling the entrance and departures of 
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multimedia source and reflectors into the application. It 
preferably also monitors the workload of the reflectors 610 
and uses the workload -related heuristics for its client admis- 
sion process. A TCP Connection Router can be implemented 
within the control server 640 to control client admission as 
well as to choose and dispatch primary and secondary (also 
called "alternate") servers for and to the client 620. 

The reflectors 610 in this example are separated into two 
sets Rl, R2, by their port numbers. When the client playback 
station CI first connects to the control server 640 requesting 
a live audio/video session, the TCP Connection Router will 
use heuristics on workload distribution to assign a reflector 
in one of the sets to the client playback station. The client 
playback station also receives the primary 650 and second- 
ary 660 entries that characterize the two sets of reflectors. If 
the client CI detects a failure of the primary server Rl, the 
client playback station CI sends a request to the control 
server 640 asking for a reflector with an alternate port 
number. Since, the alternate server is ensured to be different 
from the primary server, the client playback station can 
continue to receive the audio and video stream with minimal 
interruption. 

Alternative Embodiments 

Those skilled in the art will readily appreciate that 
equivalents, alternatives and/or extensions to the various 
embodiments can be used within the spirit and scope of the 
invention as claimed, including the following. 

Those skilled in the art will appreciate that there are a 
large number of audio/video streaming systems available in 
the market today that are adaptable to and fall within the 
spirit and scope of the present invention (see e.g., G. 
Venditto, "Instant Video," Internet World, November 1996, 
pp. 84-101). These include products such as: VDOUVE 
(available from VDOnet Corp.); STREAMWORKS X 
PLAYER (available from Xing technology Corp.); and those 
available from Vosaic LLC InterVU Inc.; and REALAUDIO 
AND REALVIDEO (available from RealNetworks Inc., 
VDOUVE, STREAMWORKS, Vosaic, and RealNetworks' 
products are based on proprietary client-server systems. 
Some systems transport their audio and/video streams using 
User Datagram Protocol (UDP/IP) connections. This unre- 
liable transport does not retransmit lost packets and is 
blocked by most firewalls unless they are specially recon- 
figured. Other systems use TCP/IP or HTTP (based on 
TCP/IP). Some products are based on Motion Picture Expert 
Group (MPEG) standards, while others uses the H.263 
standard. In general, these systems are designed to work 
over higher-bandwidth LAN connections and not at modem 
speeds. At modem speeds, the MPEG-based systems revert 
to slide-show-type video. 

According to a preferred embodiment of the present 
invention, the streaming servers are separated into two 
disjoint non-empty sets according to whether the port num- 
bers they use are even or odd. The set of streaming servers 
using odd-numbered ports are grouped into one set, and the 
set of strearning servers using even-numbered ports are 
grouped into the other set. In a first generalization, this 
invention can use any mechanisms that separate the stream- 
ing servers into two disjoint sets using their port numbers. 
This is defined by a function F for the server. As in the 
preferred embodiment, the sets created by the mechanisms 
in this generalization are preferably mutually exclusive. 
Also, both of the sets must also contain at least one func- 
tioning streaming server. As an example of this first 
generalization, two non-overlapping ranges of port numbers 
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can be used to separate the streaming servers, such as 
7000-7100 and 7200-7300. All streaming servers using port 
numbers within the range of 7000-7100 are grouped into 
one set, and all the streaming servers using port numbers 
within the range of 7200-7300 are grouped into the other 5 
set. 

A second generalization of the preferred embodiment 
relates to the placement of the primary 1,94 and secondary 
identifiers 1.96 that are used by the client agent to distin- 
guish the sets of streaming servers. This generalization io 
allows these entries to be located in other entities in the 
system described in this invention. For example, the primary 
and secondary entries for a client agent can be located at the 
control server 640. When a primary streaming server Rl 
fails, the affected Client Agents CI notify the control server 15 
640 and request to switch servers. The control server 640 can 
redirect the clients to a streaming server in the secondary set 
R2, based on their respective secondary entries. As another 
example, the secondary entry for a client agent can be 
allocated 660 at the same time as the primary 650 entry. In 20 
this case, upon detecting a failure, the client agent can 
directly access the second Streaming Set R2, without com- 
municating with the control server. 

Those skilled in the art will also appreciate that the 
present invention is not limited to conventional desktop 25 
computers and workstations and is applicable to and pro- 
vides advantages for mobile, wireless and/or so-called low- 
end devices including but not limited to hand-held devices, 
personal digital assistants and video phones that can receive 
streaming data. Furthermore the invention can be applied to 30 
the rapidly converging traditional broadcast and data 
networks, via processor-based set- top boxes and Internet 
appliances, which can bring the web to any TV via a 
standard telephone connection. By way of example only, see 
the Internet appliance sold by WebTV Networks, Inc., under 35 
the trademark WEBTV 

Now that the invention has been described by way of a 
preferred embodiment, with alternatives, those skilled in the 
art will appreciate that other alternatives, improvements and 
equivalent methods and systems can be devised within the 40 
spirit and scope of the invention as claimed. 

We claim: 

1. In a client-server system, a method of dynamically 
switching client-server connections, said method compris- 
ing the steps of: 45 

dividing a plurality of servers into two or more disjoint 
sets, wherein each set of servers includes at least one 
server; 

a client requesting a real-time data stream; 5Q 
assigning the client to both a primary server in a first set 

of servers and a secondary set of servers, in response to 

said request; 

the client receiving the real-time data stream from the 
primary server; ss 

the client detecting a failure in one or more of the 
real-time data stream and the primary server; and 

the client dynamically switching to and receiving the 
real-time data stream from a secondary server in the 
secondary set of servers, in response to said detecting eo 
step. 

2. The method of claim 1 wherein said dividing step 
further comprises the step of dividing the servers into said 
disjoint sets based on server port numbers. 

3. The method of claim 2, further comprising the steps of: 65 
said dividing step comprising the step of grouping the 

servers into two sets wherein the first set includes a 
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plurality of primary servers using odd port numbers and 
the second set includes a plurality of secondary servers 
using even port numbers. 

4. The method of claim 1 wherein the system includes a 
control server coupled to the client and the sets of servers, 
and wherein said assigning step further comprises the steps 
of: 

said requesting step comprising the client communicating 
a message to the control server, requesting a connection 
to one of the servers; and 

the control server selecting the primary server and the 
secondary server for the client. 

5. The method of claim 4, wherein said step of dynami- 
cally switching to the secondary server, further comprises 
the steps of: 

said step of dynamically switching comprising the steps 
of: 

client communicating a switch request to the control 
server; 

the control server assigning the secondary server to the 

client wherein the 
secondary server is member of a set different from the 

set of which the 
primary server is a member of; and 
establishing a new connection between the client and 

the secondary server. 

6. The method of claim 1 wherein the data stream is a 
multimedia data stream. 

7. The method of claim 6, wherein the multimedia data is 
one or more of live digital audio and video data. 

8. The method of claim 1, wherein said step of the client 
detecting a failure, further comprises the steps of: 

monitoring the data stream for data arriving out of order; 
and 

comparing the order with a threshold. 

9. The method of claim 8, wherein the data stream is 
communicated using a TCP/IP suite of protocols, said moni- 
toring step further comprising the step of: 

monitoring based on a mechanism selected from the 
group consisting of: packet numbering mechanisms 
available in TCP, and sequence numbering or time 
stamp capabilities of a Real-time Transport Protocol in 
combination with a User Datagram Protocol. 

10. The method of claim 1 wherein said each set of servers 
includes a plurality of servers and said assigning step further 
comprises the step of load-balancing client assignments to 
the primary server. 

11. The method of claim 1, wherein said step of the client 
detecting a failure, further comprises the steps of: 

measuring one or more of a received bit rate; frame rate; 

delivery rate; and sample rate associated with the data 

stream; and 
comparing the rate with a threshold. 

12. The method of claim 1, further comprising the steps 

of: 

the client communicating a switch request to a control 
server, in response to said step of detecting the failure; 
and 

the control server identifying the secondary server in the 
second set of servers based on entries maintained at a 
control server, in response to said detecting step. 

13. The method of claim 1, further comprising the steps 

of: 

the control server communicating an identifier of the 
primary server and the secondary set of servers to the 
client, in response to the client requesting the data 
stream; 
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said step of the client dynamically switching comprising said step of the client dynamically switching comprising 

the step of the client communicating a request to the the step of the client communicating a request to the 

control server to switch to the alternate server including control server to switch to the secondary server includ- 

the identifier of the secondary set of servers; and ing the identifier of the secondary server. 

the control server identifying the secondary server in the 5 23. The method of claim 16, further comprising the steps 

second set of servers in response to a client switch of: 

request a control server communicating a switch signal to the 

14. The method of claim 1, wherein the system includes client; and 

a control server coupled to the client and said each set of said step of the client dynamically switching comprising 

servers, further comprising the steps of: 1Q the step of the client communicating a switch request to 

oae of the primary server and the control server commu- the control server, in response to the switch signal; and 

nicating a switch signal to the client; and tne control server identifying the secondary server based 

said step of the client dynamically switching comprising 0 n entries maintained at the control server. 

the step of the client communicating a request to the 24. The method of claim 23, further comprising the step 

control server to switch to the alternate server, in ^ of the control server designating the secondary server as a 

response to the switch signal; and new primary server. 

the control server identifying the secondary server in the 25. The method of claim 16, wherein said step of the 

second set of servers based on entries maintained at the client detecting a failure, further comprises the steps of: 

control server. measuring one or more of a received bit rate; frame rate; 

15. The method of claim 1, further comprising the steps 20 and sample rate associated with the data stream; and 
°f : comparing the rate with a threshold. 

designating the secondary server as a new primary server. 26. A program storage device readable by a machine, 

16. In a client-server system, a method of dynamically tangibly embodying a program of instructions executable by 
switching client-server connections, said method compris- the machine to perform method steps for dynamically 
ing the steps of: 25 switching client-server connections between a primary 

identifying at least a primary server and a secondary server and a secondary server for a real-time data stream, 

server for a real-time data stream; said method steps comprising: 

a client receiving the real-time data stream from the a client receiving the real-time data stream from the 

primary server; primary server; 

the client detecting a failure in one or more of the 30 the client detecting a failure in one or more of the 

real-time data stream and the primary server; and real-time data stream and the primary server, and 

the client dynamically switching to the secondary server the client dynamically switching to the secondary server 
and receiving the real-time data stream from the sec- and receiving the real-time data stream from the sec- 
ondary server, in response to said detecting step. ondary server, in response to said detecting step. 

17. The method of claim 16, wherein the data stream is a 35 27. The method of claim 26, wherein the data stream is a 
multimedia data stream. multimedia data stream. 

18. The method of claim 17, wherein the multimedia data 28. The method of claim 27, wherein the multimedia data 
is one or more of live digital audio and video data. is one or more of live digital audio and video data. 

19. The method of claim 16, in a packet-switched network 29. The method of claim 26, in a packet-switched network 
wherein said step of the client detecting a failure, further 40 wherein said step of the client detecting a failure, further 
comprises the steps of: comprises the steps of: 

monitoring the data stream for data arriving out of order; monitoring the data stream for data arriving out of order; 

and and 

comparing the order with a threshold. 45 comparing the order with a threshold. 

20. The method of claim 19, wherein the data stream is 30. The method of claim 29, wherein the data stream is 
communicated using a TCP/IP suite of protocols, said moni- communicated using a TCP/EP suite of protocols, said 
toring step further comprising the step of: monitoring step further comprising the step of: 

monitoring based on a mechanism selected from the monitoring based on a mechanism selected from the 

group consisting of: packet numbering mechanisms 50 group consisting of: packet numbering mechanisms 

available in TCP, and sequence numbering or time available in TCP, and sequence numbering or time 

stamp capabilities of a Real-time Transport Protocol in stamp capabilities of a Real-time Transport Protocol in 

combination with a User Datagram Protocol. combination with a User Datagram Protocol. 

21. The method of claim 16, wherein the system includes 31. The method of claim 26, wherein the system includes 
a control server coupled to the client and said each set of 55 a control server coupled to the client and said each set of 
servers, further comprising the steps of: servers, further comprising the steps of: 

the client communicating a switch request to a control the client communicating a switch request to a control 

server, in response to said step of detecting the failure; server, in response to said step of detecting the failure; 

and and 

the control server identifying the secondary server based 60 the control server identifying the secondary server based 

on entries maintained at a control server, in response to on entries maintained at a control server, in response to 

said detecting step. said detecting step. 

22. The method of claim 21, further comprising the steps 32. The method of claim 31, further comprising the steps 
of: of: 

the control server communicating an identifier of the 65 the control server communicating an identifier of the 

primary server and the secondary server to the client, in primary server and the secondary server to the client, in 

response to the client requesting the data stream; response to the client requesting the data stream; 
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said step of the client dynamically switching comprising 
the step of the client communicating a request to the 
control server to switch to the secondary server includ- 
ing the identifier of the secondary server. 

33. The method of claim 26, further comprising the steps 5 
of: 

a control server communicating a switch signal to the 
client; and 

said step of the client dynamically switching comprising 
the step of the client communicating a switch request to 10 
the control server, in response to the switch signal; and 

the control server identifying the secondary server based 
on entries maintained at the control server. 

34. The method of claim 33, further comprising the step 
of the control server designating the secondary server as a 
new primary server. 

35. The method of claim 26, wherein said step of the 
client detecting a failure, further comprises the steps of: 

measuring one or more of a received bit rate; frame rate; 

and sample rate associated with the data stream; and 20 
comparing the rate with a threshold. 

36. A program storage device readable by a machine, 
tangibly embodying a program of instructions executable by 
the machine to perform method steps for dynamically 
switching client-server connections between two or more 25 
disjoint sets of servers wherein each set of servers includes 

at least one server, said method steps comprising: 
a client requesting a real-time data stream; 
assigning the client to a primary server in a first set of 

servers and a secondary set of servers, in response to 30 

said request; 

the client receiving the real-time data stream from the 

primary server; 
the client detecting a failure in one or more of the 

real-time data stream and the primary server; and 35 
the client dynamically switching to and receiving the 

real-lime data stream from a secondary server in the 

secondary set of servers, in response to said detecting 

step. 

37. The program storage device of claim 36, wherein said 40 
step of dynamically switching to the secondary server, 
further comprises the steps of: 

the client communicating a switch request to a control 
server; 

45 

the control server assigning the secondary server to the 
client, wherein the secondary server is a member of a 
set of servers different from the set of which the 
primary server is a member; and 

establishing a new connection between the client and the 5Q 
secondary server. 

38. The program storage device of claim 36, wherein said 
step of the client detecting a failure, further comprises the 
steps of: 

measuring one or more of a received bit rate; frame rate; ss 

and sample rate associated with the data stream; and 
comparing the rate with a threshold. 

39. The program storage device of claim 36, wherein said 
step of the client detecting a failure, further comprises the 
steps of: 60 

monitoring the data stream for data arriving out of order; 
and 

comparing the order with a threshold. 

40. The program storage device of claim 36, wherein the 
data stream is communicated using a TCP/IP suite of 65 
protocols, said monitoring step further comprising the step 
of: 



monitoring based on a mechanism selected from the 
group consisting of: packet numbering mechanisms 
available in TCP, and sequence numbering or time 
stamp capabilities of a Real-time Transport Protocol in 
combination with a User Datagram Protocol. 

41. The program storage device of claim 36, further 
comprising the steps of: 

the client communicating a switch request to a control 
server, in response to said step of detecting the failure; 
and 

the control server identifying the secondary server in the 
second set of servers based on entries maintained at a 
control server, in response to said detecting step. 

42. The program storage device of claim 36, further 
comprising the steps of: 

the control server communicating an identifier of the 
primary server and the secondary set of servers to the 
client, in response to the client requesting the data 
stream; 

said step of the client dynamically switching comprising 
the step of the client communicating a switch request to 
the control server including the identifier of the sec- 
ondary set of servers from which the secondary server 
is to be selected; and 

the control server identifying the secondary server in the 
second set of servers in response to the client switch 
request. 

43. The program storage device of claim 36, wherein the 
system includes a control server coupled to the client and 
said each set of servers, further comprising the steps of: 

one of the primary server and the control server commu- 
nicating a switch signal to the client; and 

said step of the client dynamically switching comprising 
the step of the client communicating a request to the 
control server to switch to the alternate server, in 
response to the switch signal; and 

the control server identifying the secondary server in the 
second set of servers based on entries maintained at the 
control server. 

44. A computer program product comprising: 

a computer usable medium having computer readable 
program code means embodied therein for dynamically 
switching client-server connections between a plurality 
of servers into two or more disjoint sets, wherein each 
set of servers includes at least one server; the computer 
readable program code means in said computer product 
comprising: 

computer readable program code means for causing a 
computer to effect a client requesting a real-time data 
stream; 

computer readable program code means for causing a 
computer to effect assigning the client to a primary 
server in a first set of servers and a secondary server in 
a second set of servers, in response to said request; 

computer readable program code means for causing a 
computer to effect the client receiving the real-time 
data stream from the primary server; 

computer readable program code means for causing a 
computer to effect the client detecting a failure in one 
or more of the real-time data stream and the primary 
server; and 

computer readable program code means for causing a 
computer to effect the client dynamically switching to 
and receiving the real-time data stream from the sec- 
ondary server, in response to said detecting step. 



06/15/2002, EAST Version: 1.03.0002 



US 6,195 : 

23 

45. A computer program product comprising: 

a computer usable medium having computer readable 
program code means embodied therein for dynamically 
switching client-server connections between a primary 
server and a secondary server for a real-time data 5 
stream, the computer readable program code means in 
said computer program product comprising: 
computer readable program code means for causing a 
computer to effect a client receiving the real-time 
data stream from the primary server; 10 
computer readable program code means for causing a 
computer to effect the client detecting a failure in one 
or more of the real-time data stream and the primary 
server; and 

computer readable program code means for causing a 15 
computer to effect the client dynamically switching 
to the secondary server and receiving the real-time 
data stream from the secondary server, in response to 
said detecting step. 

46. A client-based system for dynamically switching 20 
client connections, comprising: 
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two or more disjoint sets of real-time streaming servers, 
each set including at least one real-time streaming 
server for delivering a real-time multimedia stream; 

a control server coupled to the client and to the real-time 
streaming servers; and 

one or more client agents, coupled to the real-time stream- 
ing servers and the control server, for requesting and 
receiving and rendering the real-time multimedia 
stream; 

the control further comprising: 

control server means for processing incoming client 
agents and redirecting them to the real-time stream- 
ing server that can provide the real-time multimedia 
stream; each client agent comprising: 

client agent means for detecting a failure and dynami- 
cally switching to and receiving the real-time mul- 
timedia stream from a secondary real-time streaming 
server, said client agent means coupled to the control 
server means. 

* * * * * 
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